- Title
- An evaluation of four solutions to the forking paths problem: adjusted alpha, preregistration, sensitivity analyses, and abandoning the Neyman-Pearson approach
- Creator
- Rubin, Mark
- Relation
- Review of General Psychology Vol. 21, Issue 4, p. 321-329
- Publisher Link
- http://dx.doi.org/10.1037/gpr0000135
- Publisher
- American Psychological Association
- Resource Type
- journal article
- Date
- 2017
- Description
- Gelman and Loken (2013, 2014) proposed that when researchers base their statistical analyses on the idiosyncratic characteristics of a specific sample (e.g., a nonlinear transformation of a variable because it is skewed), they open up alternative analysis paths in potential replications of their study that are based on different samples (i.e., no transformation of the variable because it is not skewed). These alternative analysis paths count as additional (multiple) tests and, consequently, they increase the probability of making a Type I error during hypothesis testing. The present article considers this forking paths problem and evaluates four potential solutions that might be used in psychology and other fields: (a) adjusting the prespecified alpha level, (b) preregistration, (c) sensitivity analyses, and (d) abandoning the Neyman-Pearson approach. It is concluded that although preregistration and sensitivity analyses are effective solutions to p-hacking, they are ineffective against result-neutral forking paths, such as those caused by transforming data. Conversely, although adjusting the alpha level cannot address p-hacking, it can be effective for result-neutral forking paths. Finally, abandoning the Neyman-Pearson approach represents a further solution to the forking paths problem.
- Subject
- forking paths; null hypothesis significance testing; preregistration; replication crisis; sensitivity analyses
- Identifier
- http://hdl.handle.net/1959.13/1386730
- Identifier
- uon:32451
- Identifier
- ISSN:1089-2680
- Rights
- © American Psychological Association, 2017. This paper is not the copy of record and may not exactly replicate the authoritative document published in the APA journal. Please do not copy or cite without author's permission. The final article is available, upon publication, at: http://dx.doi.org/10.1037/gpr0000135.
- Language
- eng
- Full Text
- Reviewed
- Hits: 1256
- Visitors: 1488
- Downloads: 286
| Thumbnail | File | Description | Size | Format | |||
|---|---|---|---|---|---|---|---|
| View Details Download | ATTACHMENT02 | Author final version | 588 KB | Adobe Acrobat PDF | View Details Download |